Assessing Users' Subjective Satisfaction with the Information System for Youth Services (ISYS)

In this investigation, the Questionnaire for User Interaction Satisfaction (QUIS 5.5), a tool for assessing users' subjective satisfaction with specific aspects of the human/computer interface was used to assess the strengths and weaknesses of the Information System for Youth Services (ISYS). ISYS is used by over 600 employees of the Maryland State Department of Juvenile Services (DJS) as a tracking device for juvenile offenders. Ratings and comments were collected from 254 DJS employees who use ISYS. The overall mean rating across all questions was 5.1 on a one to nine scale. The ten highest and lowest rated questions were identified. The QUIS allowed us to isolate subgroups which were compared with mean ratings from four measures of specific interface factors. The comments obtained from users provided suggestions, complaints and endorsements of the system. The Information System for Youth Services (ISYS) is an on-line real-time processing system programmed in IDMS/R, a relational database program, and runs on an IBM mainframe computer. It is used by the State of Maryland's Department of Juvenile Services (DJS) as a tracking system for juvenile offenders. This system currently includes information for approximately 50,000 juveniles. The ISYS system is used at facilities throughout the state to enter and access the data needed by the Department's employees. The Questionnaire for User Interaction Satisfaction (QUIS) is a measurement tool designed to assess a computer user's subjective satisfaction with the human-computer interface and has proven reliability and validity across many types of interfaces (Chin, Diehl, & Norman, 1988). It was developed at the HumanComputer Interaction Laboratory (HCIL), University of Maryland, College Park and is currently licensed to over 76 sites both in academia and industry. The QUIS contains a demographic questionnaire, a measure of overall system satisfaction, and four specific interface factors (screen factors, terminology and system information, learning factors, and system capabilities). The QUIS was modified for use with this project. Several questions were added and questions not pertaining to ISYS were omitted. The Department of Juvenile Services contracted the Human Computer Interaction Laboratory (HCIL) to evaluate the ISYS user interface. Although ISYS has automated many of the DJS employees' tasks, it has many shortcomings which prevent it from becoming a useful tool for the employee. Because of these difficulties in using the system, employees often do not enter data about youths in a timely fashion. This makes the data in the system unreliable for those who must access and use this data for their particular job function. In our evaluation of this interface, we used the Questionnaire for User Interaction Satisfaction (QUIS) to obtain user satisfaction ratings for ISYS. We examined these ratings in order to determine what areas of the system needed improvement, which aspects were satisfactory and to suggest possible reasons why these aspects were rated as higher or lower. The Department of Juvenile Services consists of three major employee divisions, Administrative, Field Services and Residential Services. The employees within these divisions use the system for a variety of job tasks. For example, employees in Field Services, which consists mainly of Juvenile Counselors, enter data about the youths. While some of the information Field Services employees place in the system may be useful to them, the majority of the data is accessed by others such as Administrative employees who must gather information for statistical or financial purposes. Because of these variations in system use, we predicted differences in satisfaction measures between the employee divisions. Questions were added to the demographic portion of the QUIS in order to identify these subgroups. A separate section for the QUIS was created to answer questions about ISYS training. The additional ISYS training section was requested by the Department of Juvenile Services to look at the amount and quality of the training given to employees. For the most part, their goal was to document the inadequacy of training. Many of the results from our analyses of the QUIS ratings and comments made by users are presented in this report. It was our goal in this project to make recommendations to the Department of Juvenile Services for improvements to ISYS. For the most part, these recommendations were made after many hours of observation and informal interviews with employees. The QUIS results from this investigation were partly used to confirm the findings made from the observations and give a scientific backing to the recommendations. This report will not center on a comparison of observational data and QUIS data. The goal in this paper is to document the use of the QUIS in the evaluation of a system. We used ratings from the questionnaire to identify the strengths and weaknesses of the system. User's comments were used to acknowledge specific problems of the system. In our discussion, we will show how the results from the QUIS allowed us to give a more in depth look at the system and infer why specific aspects of the system rated as they did.